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CONFIDENTIAL 



Background 



Field of Invention 

The present application relates to electronic messaging applications, specifically 
to electronic messaging systems invoked within the context of an application. 



Electronic computer games are becoming very sophisticated interactive 
applications. One user's experience in a game may be very different than other users 1 
experience because of the level of sophistication and variety of game experiences 
provided even in a single game. For example, while playing an electronic game, players 
10 may overcome obstacles, visit different lands, meet and talk to different people, and thus 
different players may have completely different experiences while playing the same 
game. Much of the entertainment value of playing electronic games is derived from 
discussing the experience with other players of the game and other friends who may not 
be players of the game. However, currently the only method to relate an individual 



15 gaming experience in any significant detail is to relate the experience orally. 

Alternatively, sophisticated computer users may print-out a screen shot of a particular 

game image to send to others. 

However, screen shots are very difficult to use. Locating the image file after 

capture requires a user with some computer sophistication, and sending the image file 
20 require the use of a third party electronic mail program. Moreover, third party electronic 

mail programs often cannot be activated while a game is being executed, and therefore in 

order to send the message the user must exit the game, locate the screen shot image file, 
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execute the mail program, create a new message, and then attach the image to the 
message. Users who do not have third party electronic mail programs cannot use the 
screen shot method at all, and for most other users the screen shot method is too 
cumbersome to be a viable solution to this problem. Screen shots are also limited to 
5 images, and cannot capture other types of media produced by the game. Further, screen 
shots cannot easily be modified or customized to suit user preferences. For example, 
screen shots are unable to distinguish between desired content displayed on a screen and 
undesirable content. 

Therefore, a system, method and apparatus are needed in which electronic 

10 messages can be sent by a user from within a game or other application that captures the 
experience of a moment within the game, provides the ability to transmit that experience 
to others, and provides a user-friendly format to transmit the experience, without 
requiring exiting the game, using a third party electronic mail program to send the 
message, and does not require a user to possess an advanced level of computer 

15 sophistication. 

Summary of invention 

In accordance with the present invention a composite message is generated 

representative of an application experience. This composite message is formed by 

combining a user input selecting multimedia information provided by an application and 

20 message information received from a message form generated from within the 

application. The composite message is then sent to the intended recipient(s) specified by 

the user through a network connection. In a preferred embodiment, a composite message 



3 




CONFIDENTIAL 



is created by any or all of the following: capturing a current screen image, linking an 
associated audio file to the message, removing extraneous graphical data of an 
application and/or user interfaces from the image, scaling the image to desired display 
size, adding corporate logo or other identifying information to the message, and receiving 
5 text to be transmitted to a remote recipient. 

In a further embodiment, the electronic mail address of the sender is automatically 
retrieved and placed in a message form generated internally in the application module, the 
recipient's name and address are received, and the address information, message text, and 
multimedia information are displayed to the sender to allow the sender to decide whether 

10 or not to send the composite message. Upon sending the composite message, the normal 
execution of the application is resumed. In a further embodiment, the system also 
determines if the user is currently connected to a network, and initiates the connection if 
the user is not connected. The system can also notify the sender when the recipient has 
received the message during the execution of the application. 

15 Thus, the present invention provides an experience representation that does not 

require exiting the application from which the experience is desired to be captured, does 
not require a third party electronic mail program to send the message, and requires little 
computer sophistication on the part of the user. Moreover, the present invention 
combines customizable, alterable multimedia information with a message in a user- 

20 friendly format and delivers the message quickly and reliably to an intended recipient. 

In an alternate embodiment, the present invention notifies a predetermined third 
party of the identity of the recipient of the composite message. The third party can use 
the information to market the application or other products/services, to the recipient if the 
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recipient is not currently an owner of the application, or purchaser of the other products 
or services. 

Brief descripton of the drawings 

Fig. 1 is a block diagram illustrating a network connecting a sender of a 

5 composite message to a recipient. 

Fig. 2 is a flowchart illustrating generating a composite message. 

Fig. 3a is a flowchart illustrating an embodiment of capturing multimedia 

information. 

Q 

^0 Fig. 3b is a screen shot of an image being captured. 

10 . Fig. 3c is a screen shot of an image being captured with extraneous information 

m removed. 

a; Fig. 4a is a flowchart illustrating an embodiment of generating a message form. 

; ~ Fig. 4b is a screen shot of a first stage of a message form being created. 

S Fig. 4c is a screen shot of a second stage of a message form being created. 

15 Fig. 5 is a flowchart illustrating an embodiment of creating a composite message. 

Fig. 6 is a flowchart illustrating an embodiment of sending a composite message. 
Fig. 7 is an illustration of receiving a composite message through a conventional 
e-mail program. 

Fig. 8 is a flowchart illustrating a method of providing targeted advertisements. 

20 
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Detailed description of the preferred embodiments 

Fig.l is a block diagram illustrating a system 128 connecting a sender of a 

composite message in accordance with the present invention to a recipient through 

network 124. In accordance with such system 128, there is provided computers 100 
5 communicatively coupled by a network 124, such as the internet, to one or more 

computer servers 132. Although two computers 100 are shown, there is no physical limit 

to the number of computers 100 that may be connected to the network 124 at one time. A 

computer 100(1) is used by a sender to execute an application/messaging module 104. 

The application messaging module 104 includes an application module 105 and a 
10 composite message generation module 106. The application module 105 is preferably an 

electronic game application; however, the application module 105 can be any application 

providing images or other media that a sender wishes to send to a remote recipient. 
The computer 100 further comprises a WinSock/Winlnet component 108 or 

equivalent networking interface, a TCP/IP or other netware protocol subsystem 1 12, a 
15 network hardware and driver module 1 12, or alternatively a modem and modem driver 

120, or both. The network hardware and driver module 116 provides an Ethernet 

connection to server 132. 

The modem and driver 120 provides a telephone connection to server 132. In a 

preferred embodiment the computer 100 may be implemented on an Intel-based computer 
20 operating under Microsoft Windows 3.1, Windows95, or WindowsNT operating system, 

or a SPARC-based computer operating under UNIX, or any other equivalent devices. 

The computer 100 is preferably capable of generating graphical representations, typically 
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through the use of an internal graphics or rendering pipeline and associated hardware as is 
known in the art. 

The computer 100(2) is similarly configured as computer 100(1) except that the 
application/messaging module 104 does not have to be present in computer 100(2). 
However, computer 100(2) has a electronic mail messaging program 145 or equivalent 
(e.g. browser based email reader - such as Hotmail (TM)) to receive and view electronic 
mail messages in the conventional manner. In a preferred embodiment, the electronic 
mail messaging program is capable of reading e-mails with embedded Hypertext Markup 
language (HTML). 

Figure 2 is a flowchart illustrating generating a composite message in accordance 
with the present invention. First, a composite message generation command is received 
200 during execution of the application module 105 on computer 100(1), invoking the 
composite message generation module 106 and, in a preferred embodiment, pausing the 
execution of the game. This allows the sender to generate a composite message from any 
point during the execution of the application module 105 by activating the control. 
Alternatively, a key of the keyboard of the computer 100(1) is used to begin execution of 
the composite message generation module 106, for example, by pressing a function key 
or other key combination. Next, multimedia information is captured 204. Multimedia 
information, as discussed below, includes images, audio files, or any other media 
provided by the application module 105 and capable of being transmitted over the 
network 124. An example of an image 350 is shown in Figure 3b. 

Then, a message form for containing the message is generated 208. The message 
form is used to receive message information and message text from the sender, including 





CONFIDENTIAL 



recipient data such as the recipient's email address and name. One example of a message 

form 400 is shown in Figure 4a. The message information may be text as shown in Fig. 

4a or 4b or in an alternate embodiment may be recorded audio, video or a combination of 

any of these. Next, a composite message is created 212. The composite message 

combines the message information from the message form and the multimedia 

information into a single display. Figure 7 illustrates one format of a composite message 

700, an electronic postcard. However, any other message format in which multimedia 

and text information can be displayed is considered to be within the scope of the present 

invention. In one embodiment, the composite message itself is used to receive message 

information directly from the user without the intermediate message form 400. 

Then, the composite message is sent 216 to the recipient via network 124. The 

recipient, as discussed above, does not require a copy of the application module 104 to 

view the message, but need only have a conventional electronic mail package supporting 

audio/image tracks. The recipient may now view the composite message 700 containing 

the multimedia and message information on the recipient's computer 100(2), as shown in 

Figure 7. This allows the sender to effectively share the sender's experience with the 

recipient. The recipient may also save or print out the composite message 700. 

Responsive to sending the message, execution of the game is resumed. In one 

embodiment, the sender is displayed a notification within the game that the sent message 

has been received, as is accomplished in conventional e-mail applications, but as the 

messaging is part of the application module, display of such message does not interrupt or 

hinder the execution of the game. Thus, by incorporating the composite message 

generation with an application module, the user is able to send composite messages 
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seamlessly from within an executing application without exiting the application or 
hindering the gameplay of the application if the application is an electronic game. 

Figure 3 a is a flowchart illustrating an embodiment of capturing multimedia 
information in accordance with the present invention. First, in a preferred embodiment, 
5 the composite message generation module 106 determines 302 whether extraneous user 
information elements or data is overlaying the image currently being displayed by the 
application module 105. Typically, application modules 105 display user interfaces or 
other graphical information 300 as part of the game (generically application) to 
communicate information to the player, as shown in Figure 3b. In accordance with the 

10 present invention, any user interfaces or other extraneous graphical elements 300 
generated as part of the application module 105 itself or generated by third party 
software, for example, the operating system or other applications that may be executing 
in the background, are removed 304 from the image prior to capture. This provides a 
complete, clear image for transmission as shown in Figure 3c. As the user interfaces are 

15 typically rendered as one of the last layers by the rendering pipeline, in a preferred 

embodiment, the application module 105 instructs the pipeline to skip the user interface 
render stage for the image to be captured. 

Next, the current image being displayed by the application module 105 is captured 
308. This can be accomplished by several methods, for example, by retrieving data 

20 stored in a display buffer for the computer at the time the composite message generation 
button is selected and then storing the retrieved data in memory. In an embodiment in 
which user interface elements are removed, the composite message generation module 
106 determines 312 whether a user interface was removed previously. This may also be 
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accomplished by several methods, for example, by setting a status indicator indicating 
that a user interface was removed upon removal of the interface, and checking the status 
indicator after the image has been captured to determine whether a user interface should 
be reinstated. If a user interface was removed, the user interfaces are redisplayed 316, 
5 preferably by instructing the pipeline to resume rendering the user interface stage. 

Next, the composite message generation module 106 scales 320 the captured 
image to a size appropriate for the composite message to be transmitted. In a preferred 
embodiment, the image is scaled down to a resolution 320x240 from the current screen 
resolution using conventional graphics techniques. In one embodiment, an interface is 

10 provided allowing the user to drag a window on the screen surface, allowing the user to 
choose the portion of the screen to be placed in the postcard. This embodiment provides 
greater user control over the content of the composite message. The selected portion is 
then resized to fit an appropriate size for the composite message. 

Then, the composite message generation module 106 retrieves 324 or captures 

1 5 audio information to accompany the image. The audio information can be a default audio 
clip stored by the application, for example, indicating a unique sound of the application 
or identifying the application, or the audio information can be an audio file that is 
currently being executed at the time of image capture. This embodiment increases the 
ability of the sender to capture the experience of the game or application module 105. 

20 Additionally, in one embodiment, a sender is given the opportunity to record an audio file 
to accompany the image, for example, recording a greeting by capturing the sender's 
voice through a microphone provided with the computer 100 or the 
application/messaging module 104. 

10 
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Figure 4a is a flowchart illustrating an embodiment of receiving message 
information in accordance with the present invention. First, a message form 400 is 
generated 402. The message form 402 is an interface generated through conventional 
means that provides a "mechanism" or "means" through which the sender can enter the 
5 message information of the composite message. As discussed above, message 

information is typically the recipient's name, e-mail address, and a message text, as well 
as the sender's name and e-mail address. Figures 4b and 4c are screen shots of a 
preferred message form 400 within which message information can be entered by the 
sender, however, other forms for entry of information are considered to be within the 

1 0 scope of the present invention. 

The message form 400 provides a entry field 416 to allow the sender to enter the 
sender's e-mail address. However, in a preferred embodiment, the sender's e-mail address 
is retrieved 404 from information stored by the application/messaging module 104, for 
example, when the sender registers the product with the manufacturer, or in an initial set- 

15 up screen for the application/messaging module 104. The composite message generation 
module 106 also displays an entry field 420 allowing the composite message generation 
module 106 to receive 408 the recipient's name and address. In one embodiment, the 
composite message generation module 106 displays a list of recipients to whom the 
sender has previously sent messages, and allows the sender to choose a recipient from the 

20 displayed list. The composite message generation module 106 also displays an entry 
field 424 allowing the composite message generation module 106 to receive message 
text. The message text is simply the personal text greeting the sender wishes to 
accompany the multimedia information. 
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Figure 5 is a flowchart illustrating an embodiment of creating a composite 
message in accordance with the present invention and is described with respect to Figure 
4c and 7. First, the composite message generation module 106 positions 500 the captured 
image 500 within the message form. As shown in Figure 7, in an electronic postcard 
embodiment of the present invention, the composite message generation module 106 
displays the image 300 in an image half 708 of the message 700, as in a traditional 
postcard. Next, the composite message generation module 106 displays 504 message text 
in the composite message 700, and as shown in Figure 4c, in the electronic postcard 
embodiment, the message text is displayed in the second, text, half 704 of the postcard. 
Then, the composite message generation module 106 displays 508 the sender's address in 
a sender information area 712 and displays 512 the recipient's address in a recipient data 
area 716. If an audio file is to be attached, the composite message generation module 106 
attaches 516 the audio file to the message form and displays an audio icon on the message 
form indicating the presence of the audio file. Then, the composite message generation 
module 106 adds 520 additional information 720 to the message form. For example, in 
the embodiment shown in Figure 4c, a corporate logo is provided for display on the 
message form providing a means for advertising to the manufacturer of the application 
module 105. Additionally, in this embodiment, a postmark graphic and a stamp image are 
provided to complete the "postcard" image. Any other information can also be added, for 
example audio files or personal images selected by the sender, or other graphics to create 
a different form within which to present the sender's message and the selected multimedia 
information. 

12 
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Figure 6 is a flowchart illustrating an embodiment of sending an composite 
message in accordance with the present invention. First, after the sender completes entry 
of data into the message form, the composite message generation module 106 displays 
600 the composite message to the sender. In this embodiment, the option of sending the 
5 message is displayed 604, or canceling the message. If the sender does not want to send 
the message, the composite message generation module 106 returns 608 control to the 
application module 105, and the execution resumes from the point at which the sender 
initiated the composite message generation module 106 originally. If the sender does 
want to send the message, the image is converted 608 to a compressed format, for 

10 example, in JPEG, to prepare the image for transmission. Any other means of 
compression could also be used to compress the image suitably for transmission. 

Next, the message information is restructured 612 to comply with- an electronic 
mail protocol. In one embodiment, the basis for the email is an HTML stub file which 
contains an HTML <img> tag for the image data. The ASCII e-mail message is created 

15 by wrapping the HTML in MIME (multipurpose internet mail extension) format. Any 
other method for transmitting text and non-text files via Internet e-mail may also be used 
in accordance with the present invention. Using the MIME method, the composite 
message is encoded using base 64 coding on the transmission end, and is decoded using 
the inverse process at the receiving end. A MIME header is added to the composite 

20 message that indicates the type of data within the message and the encoding method used. 

In this embodiment, a MIME message with multipart/related content type is used, having 

subparts of text/html and image/jpeg to accommodate the text and image portions of the 

composite message. The stub HTML code becomes the text/html part of the MIME 
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encoded message, and, as discussed above, the captured image is encoded using base64 
encoding, and becomes the image/jpeg part of the MIME encoded message. Other 
content allowable in HTML could also be included as is known to those of ordinary skill 
in the art. 

The composite message generation module 106 then displays 616 a message 
asking the sender whether the message should be sent now or later. If the sender chooses 
to defer the message, the completed e-mail is stored 618 until the next time the sender is 
connected online. The e-mail may be stored on the sendees computer 100(1), in memory, 
on a local or external hard drive, or, in a preferred embodiment, stored in an outbox of the 
sender and queued for delivery the next time the sender activates the messaging 
application. Any other suitable means for storing information could also be used. 

If an immediate send is requested, then the composite message generation module 

106 determines 624 whether the sender has an existing online connection. This is 

accomplished by querying the operating system of the computer 100(1), or by other 

conventional means. If the sender is not connected, the composite message generation 

module 106 provides the sender the option to connect 626 to a server 132. In accordance 

with the present invention, the connection is made without exiting the 

application/messaging module 104. In an alternate embodiment, if there is no connection 

to the server 132, the message is automatically stored for later delivery. For example, the 

message could be stored in the outbox, and when the sender is connected to the server 

132 next, a reminder message is displayed to remind the sender that an unsent message is 

in the sender's outbox. If the connection is made or if the connection was already 

existing, the message and any stored messages are sent 628. In one embodiment, the 
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composite message generation module 106 disconnects the sender after sending the 
message. Then, the composite message generation module 106 restores the execution of 
the application module 105 at the point at which the sender initiated the message sending 
feature in accordance with the present invention. 
5 The composite message is sent through a TCP/IP connection to a server 132, 

where it is relayed to the recipient in the conventional manner. Any mail server that 
implements SMTP or other compatible format can receive the composite message. If the 
message is transmitted over a local-area or other type of network, other connections may 
be used to transmit the message. 

10 Figure 7 is an illustration of a composite message received through a conventional 

e-mail program. The recipient is not required to have the application/messaging module 
104 stored on their computer 100(2); all the recipient should have is an email viewer 
capable of reading emails with embedded HTML. In an alternate embodiment the 
composite message is sent as an attachment to an e-mail, for example, as an attached .gif 

15 file. 

In one embodiment, as shown in Figure 8, messages transmitted by users of the 

application/messaging module 105 are transmitted 800 to a specific server maintained by 

a third party interested in providing marketing or advertising services. In this 

embodiment, the third party logs 804 in the electronic mail address of the recipient of the 

20 message. Once stored, the third party compares 808 the recipient's identifying 

information with a database of registered owners of the application/messaging module 

104. If the third party determines 812 that the recipient is not a registered owner, the third 

party sends 816 marketing information to the recipient's electronic mail address. This 
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allows the third party to target customers who are likely purchasers of the 

application/messaging module because the recipient is acquainted with the product 

through the sendees composite message, and the recipient may have similar interests as 

* 

the sender. Therefore, the composite messaging system also provides a powerful targeted 
advertising feature for the benefit of the manufacturer of the application/messaging 
module 104. 

In one embodiment, the manufacturer of the application/messaging module 104 
maintains the server 132; however, in an alternate embodiment, the server 132 is 
maintained by a third party who provides marketing and advertising services to the 
manufacturer of the application module 105. Alternatively, the manufacturer maintains 
the server 132, but forwards composite messages to the third party to have the third party 
evaluate and provide advertising if appropriate. In yet another embodiment, the 
composite messages are generated by the manufacturer of the application/messaging 
module 104 or a third party working with the manufacturer, and are sent as 
advertisements to targeted customers. 

Thus, in accordance with the present invention, users of an application module 
105 are allowed to interact with their fellow users and share the in-game or application 
experience. The composite messages are as unique as each user's gaming experience. 
This feature allows a community to be built centered around a specific application 
module 105 and allows the manufacturer to establish and maintain brand differentiation 
and brand loyalty to an application module 105 by providing the interactive snapshots of 
the application module 105 to other users and potential users of the application module 
105. 
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